Transforming locations in a spherical image viewer

ABSTRACT

A method and apparatus are provided for transforming data provided in a spherical format. A spherical format is created of an image obtained by a camera, the spherical format comprising a notional sphere that has a centre corresponding to the position from which the image was obtained by the camera. A first surface represented in the image had a first orientation and was at a first distance from the camera when the image was obtained. A selected point in said spherical format is obtained, which is defined by spherical coordinates consisting of a yaw angle and a pitch angle defining a line from the centre of the sphere. A plane is identified that has the first orientation and is at the first distance from the centre. A location in a Cartesian coordinate system is calculated where the plane intersects with the line, wherein two of the axes of the Cartesian coordinate system are parallel to the first plane. Thereby, the position of the point on the first surface is identified. This location, and other locations corresponding to to other selected points, may be used to transform data in the spherical image for display.

CROSS REFERENCE TO RELATED APPLICATIONS

This application represents the first application for a patent directedtowards the invention and the subject matter.

BACKGROUND OF THE INVENTION

The present invention relates to a method of producing output image datarepresenting a Cartesian two-dimensional floor plan from spherical inputdata of a room with a floor and walls.

The present invention also relates to an apparatus for producingeye-readable graphical presentations of floor plans.

The present applicant has commercially exploited a system in the UnitedKingdom for displaying a virtual map, used predominantly for providingimmersive and interactive images of the interiors of rooms within housesand flats etc. Projections of images are generated and indications of afirst position in a first image are obtained along with an indication ofa second position within a second image. Within each rendered image, anelement is present corresponding to the position of the other image suchthat, when operating within an immersive environment, it is possible totraverse from room to room.

In the implementation, original image data is recorded asomnidirectional input image data and sophisticated processing techniquesare performed, as detailed herein, to provide the immersive environment.However, the image when viewed in a spherical format provides noinformation regarding the physical space mapped by the image, as allpoints are represented as on the internal surface of a notional sphere,and having an equal distance from the central viewing point. Theapplicant has therefore identified a requirement for generatingcoordinates in a Cartesian system from the two spherical coordinatesprovided by the spherical format.

BRIEF SUMMARY OF THE INVENTION

According to a first aspect of the invention, there is provided a methodof transforming data provided in a spherical format, comprising thesteps of, at a processor: creating a spherical format of an imageobtained by a camera, wherein said spherical format has a centre thatcorresponds to the position from which said image was obtained by thecamera, and wherein a first surface represented in said image had afirst orientation and was at a first distance from the camera when theimage was obtained; obtaining a selected point in said spherical format,defined by spherical coordinates consisting of a yaw angle and a pitchangle defining a line from said centre; obtaining said first orientationand said first distance; identifying a plane having said firstorientation that is at said first distance from said centre; calculatinga location in a Cartesian coordinate system where said plane intersectswith said line, wherein two of the axes of said Cartesian coordinatesystem are parallel to said first plane, thereby identifying theposition of the point on said first surface; and using said location totransform data in said spherical image for display.

According to a second aspect of the invention, there is providedapparatus for identifying a location on a surface represented in aspherical image, comprising a processor and a memory, wherein saidprocessor is configured to: create a spherical format of a first imageobtained by a camera, wherein said spherical format has a centre thatcorresponds to the position from which said image was obtained by thecamera, and wherein a first surface represented in said first image hada first orientation and was at a first distance from the camera when theimage was obtained; obtain a selected point in said spherical format,defined by spherical coordinates consisting of a yaw angle and a pitchangle defining a line from said centre; retrieve from memory said firstorientation and said first distance; identify a plane having said firstorientation that is at said first distance from said centre; calculate alocation in a Cartesian coordinate system where said plane intersectswith said line, wherein two of the axes of said Cartesian coordinatesystem are parallel to said first plane, thereby identifying theposition of the point on said first surface; and use said location totransform data in said spherical image for display.

According to a third aspect of the invention, there is provided anon-transitory computer-readable medium with computer executableinstructions stored thereon, wherein said instructions configure aprocessor to perform the method of: creating a spherical format of animage obtained by a camera, wherein said spherical format has a centrethat corresponds to the position from which said image was obtained bythe camera, and wherein a first surface represented in said image had afirst orientation and was at a first distance from the camera when theimage was obtained; obtaining a selected point in said spherical format,defined by spherical coordinates consisting of a yaw angle and a pitchangle defining a line from said centre; obtaining said first orientationand said first distance; identifying a plane having said firstorientation that is at said first distance from said centre; calculatinga location in a Cartesian coordinate system where said plane intersectswith said line, wherein two of the axes of said Cartesian coordinatesystem are parallel to said first plane, thereby identifying theposition of the point on said first surface; and using said location totransform data in said spherical image for display.

Embodiments of the invention will be described, by way of example only,with reference to the accompanying drawings. The detailed embodimentsshow the best mode known to the inventor and provide support for theinvention as claimed. However, they are only exemplary and should not beused to interpret or limit the scope of the claims. Their purpose is toprovide a teaching to those skilled in the art.

Components and processes distinguished by ordinal phrases such as“first” and “second” do not necessarily define an order or ranking ofany sort.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 shows an environment in which the invention may be used;

FIG. 2 shows a floor plan of a space to be mapped;

FIG. 3 is an illustration of a virtual map displayed on a tablet shownin FIG. 1;

FIG. 4 details steps carried out to create and display a virtual map;

FIG. 5 illustrates a spherical camera photographing a room;

FIG. 6 illustrates a spherical photograph taken by the camera shown inFIG. 5;

FIG. 7 is a diagram of a server shown in FIG. 1;

FIG. 8 shows steps carried out by a processor shown in FIG. 7;

FIG. 9 shows the contents of a memory shown in FIG. 7;

FIG. 10 details steps carried out during FIG. 8 to carry out serverinstructions;

FIG. 11 is a diagram of a computer shown in FIG. 1;

FIG. 12 shows steps carried out by a processor shown in FIG. 11 tocreate a virtual map;

FIG. 13 shows the contents of a memory shown in FIG. 11;

FIG. 14 illustrates an interface displayed on a display shown in FIG.11;

FIG. 15 is a representation of a layout of map data;

FIG. 16 details steps carried out by the processor shown in FIG. 11 toupload images;

FIG. 17 illustrates the generation of a circular projection from animage;

FIG. 18 details steps carried out during FIG. 16 to create the circularprojection shown in FIG. 17;

FIG. 19 illustrates calculations carried out during FIG. 18;

FIG. 20 illustrates further calculations carried out during FIG. 18;

FIG. 21 details steps carried out during FIG. 18 to modify the alphachannel;

FIG. 22 details steps carried out by the processor shown in FIG. 11 tolink images;

FIG. 23 illustrates a database structure that stores a virtual mapdefinition;

FIG. 24 is a diagram of a tablet computer shown in FIG. 1;

FIG. 25 shows steps carried out by a processor shown in FIG. 24 to viewa virtual map;

FIG. 26 shows the contents of a memory shown in FIG. 24;

FIG. 27 details steps carried out during FIG. 25 to respond to customerinput on a virtual map;

FIG. 28 details processes included in the viewer instructions shown inFIG. 13;

FIG. 29 shows a user interface displayed on the display shown in FIG.11;

FIG. 30 illustrates a three-dimensional model, established usingspherical coordinates;

FIG. 31 shows the derivation of equations used in the method hereindescribed;

FIG. 32 illustrates a camera obtaining an image;

FIG. 33 details a camera pose compensation process shown in

FIG. 28;

FIG. 34 shows a user interface during the steps carried out in FIG. 33;

FIG. 35 details steps carried out during FIG. 33 to estimate camera poseangles;

FIG. 36 details steps carried out during FIG. 35 to transform sphericalcoordinates into Cartesian coordinates;

FIG. 37 details a floor plan creation process shown in FIG. 28;

FIG. 38 shows a data structure storing a cycle graph;

FIGS. 39 to 41 show a user interface during the steps carried out inFIG. 37;

FIG. 42 details steps carried out during FIG. 37 to obtain coordinatesof corners of a room in an image shown in FIG. 29;

FIG. 43 details a cursor movement process shown in FIG. 28;

FIG. 44 details steps carried out during FIG. 43 to calculate a point onan opposite plane to the cursor;

FIG. 45 details steps carried out during FIG. 43 to render guidelines;

FIG. 46 shows a user interface during the steps carried out in FIG. 37;

FIG. 47 details steps carried out during FIG. 37 to render a floor plan;and

FIG. 48 shows a floor plan of a building produced by the steps of FIG.37.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION FIG. 1

An environment in which aspects of embodiments described herein may bedeployed is shown in FIG. 1. A first web server 101 hosts a firstwebsite, for a first entity, and a second web server 102 hosts a secondwebsite for a second entity. In accordance with proposals set by theapplicant elsewhere, both said websites have at least one virtual mapembedded therein. Customers 103, 104, and 105, browsing internet 106 onany computing device, such as a desktop computer, laptop, tablet, mobilephone, or any other device, view the virtual maps, therefore theembedded maps are retrieved from virtual map server 107 and returned tothe browsing customers over the internet 106.

The present applicants have deployed this proposal for the purposes ofproviding details of houses and apartments etc, to allow prospectivecustomers to obtain detailed information without attending a site visit,or before such a visit. The virtual maps provide an immersive experiencefor customers as they virtually move from room to room in a virtualenvironment. However, in presentations of this type, it would be usefulto be able to identify the location of certain elements that can be seenwithin the image. An example is the generation of a floor plan byidentifying the location of corners and walls of a room. The aforesaidproposal for generating a sophisticated virtual tour requires minimalskill and labour. It is therefore undesirable to then be required toinstruct a draughtsman to produce the, more mundane, two dimensionalimages. The present invention therefore seeks a technical solution forgenerating Cartesian location data for elements visible in the sphericalimages.

Users 108 and 109 create and maintain the virtual maps by communicatingover internet 106 with virtual map server 107. Again, users 108 and 109may be using any type of computing device. The virtual maps describedherein and expanded in the previous proposal can be created on anythingfrom a smartphone to a high-powered desktop publishing computer.

For the purposes of clarity, terminology will be used consistentlyherein, in that users create virtual maps and floor plans; and customersaccess them. In the embodiment described herein, the virtual maps andfloor plans are hosted on the virtual map server 107 and the relevantwebsites contain an embedded link. However, the virtual maps could alsobe hosted on the individual web servers or in some other location.

FIG. 2

A ground floor of a house 201 for which a virtual map and a floor planis to be created is shown in FIG. 2. The house 201 has a front garden202, a back garden 203 and the rooms on this floor of the house includean entrance hall 204, a study 205, a cloakroom 206, a bathroom 207, anda living room 208. Doors connect each of these rooms. In addition, thereis a front door to the front garden 202, while the back garden 203 isaccessed by sliding doors in the living room 208. Stairs 209 lead to thefirst floor.

In known virtual tours, an architectural layout of the house would becombined with photographs in order to render a tour. However, theinvention described herein provides a method of linking a set ofpanoramic photographs. A panoramic photograph is a rectangularphotograph that represents a 360° view of a space. In the embodimentdescribed herein, omnidirectional or spherical photographs are used,which also show the floor and ceiling. However, in other embodiments,basic panoramic photographs could be used.

Photographs are taken of each room to be included in the tour. In somesituations, two photographs of a room may be appropriate. Thus,returning to the example in FIG. 2, one photograph is taken of each offront garden 202, back garden 203, study 205, cloakroom 206, bathroom207 and living room 208. Two photographs are taken of entrance hall 204,because it is an L-shape.

The next stage involves establishing connection points between thephotographs. Thus, for example, point 210 indicates the door from theentrance hall into the study. This point would be marked by the userboth on the photograph of the study 205 and on the photograph of theentrance hall 204. All the connection points marked with a cross on thisfigure would be similarly defined as connections between twophotographs. Point 211, which is at the bottom of the stairs, would belinked with a photograph of the landing on the first floor, should theuser wish to add the first floor to the virtual tour.

The set of images and the connection point information is all that isnecessary to create a three-dimensional virtual map, which can beenhanced by identifying locations in Cartesian coordinates of elementsvisible in the image. The images and the connection points can berendered into a viewable tour of the house which allows users topanoramically view each room before moving into the next.

FIG. 3

An illustration of a virtual map or tour is shown in FIG. 3, asdisplayed on a tablet. This is identified as the device 103 used by acustomer in FIG. 1. The interface includes image viewer 301 and toolbar302. The image currently being displayed within image viewer 301 is ofthe study 205. Connection point 210 is visible as a target-shaped icon303 on door the 304. Text 305 indicates that this is connection pointmarks the door to entrance hall 204.

The customer may move the panoramic image around in any way appropriateto the device being used. Thus, if the device has a touchscreen ortouchpad, the image may be intuitively moved around using a finger. On acomputer, the mouse may be used or the keyboard. On a handheld device,accelerometers contained within the device could be used to move theimage dependent upon the movement of the device. Alternatively, arrowson the toolbar 302 may be used.

When the customer has viewed as much of this room as is required, icon303 may be selected, in any way appropriate to the device being used, inorder for a panoramic image of entrance hall 204 to be displayed. Onreviewing FIG. 2, it can be seen that in this next image of the entrancehall, five icons representing connection points would be displayed,allowing the customer to choose to view the other half of the entrancehall, to move back into the study, to view the bathroom or the livingroom, or to go upstairs. In each case, wherever the customer selects aconnection point, the image of the next room is orientated from thepoint of view of the corresponding connection point, i.e. as if theviewer were standing directly in front of the connection point. Thus,the customer has the experience of simply stepping through the door orother connection point. Consequently, this system of the previousproposal provides a customer with an intuitive and seamless method ofmoving through the displayable space.

Although doorways represent natural connection points between images,sometimes connection points may be in the middle of a space, such asconnection point 212 that connects two images of the entrance hall 204.In a large open plan space, for example a gym or a garden, mostconnection points would be of this type. However, if the user wishes tomake a floor plan corresponding to this virtual map, as will bedescribed further with respect to FIGS. 38 to 48, all connection pointsmust represent a connection in a wall and not in the interior of aspace.

FIG. 4

A flowchart for the workflow of the creation, enhancement and viewing ofa virtual map or tour is shown in FIG. 4. At step 401 a user takesspherical photographs of a space to be mapped. As previously described,panoramic images may be used, but in the preferred embodiment sphericalphotographs are used that allow the customer to see every part of thespace.

At step 402, the user uploads images to their computing device, and atstep 403 the device creates and displays projections of these imagesthat enable the user to create connections between them, which the userdoes at step 404. The images and connections together define the virtualmap, which is rendered and stored at server 107 at step 405. These stepsare performed in accordance with the previous proposal.

At step 406, the virtual map may be enhanced by identifying locations inCartesian coordinates of elements visible in said images. Thisenhancement may also take place if the user has not carried out steps403 to 405 to create a virtual map, as many of the enhancement methodsare suitable for use in any spherical viewer, as well as the onepresently described.

At step 407 customers access the virtual map on server 107, usually viawebserver 101 or 102.

FIG. 5

A spherical camera 501, mounted on a tripod 502, taking a sphericalphotograph of study 205, is shown in FIG. 5. A typical spherical camerahas two cameras back-to-back, with a fisheye lens on each. However, anycamera that obtains a spherical photograph can be used. In thisembodiment the images are all in two dimensions but three dimensionalimages can be used.

FIG. 6

An example of a spherical photograph of the study 205, taken by thespherical camera 501, is shown in FIG. 6. It is an equirectangularprojection, which is the typical output of a spherical camera. Theceiling and the floor are stretched, but the walls are mainly inproportion. This is the format of the images that may be uploaded andused as the basis of the virtual map, and that may be enhanced, forexample by adding a floor plan. In addition, if the camera 501 is notperfectly aligned with the room, the image can be enhanced to identifythe rotational compensation that should be applied to the image.

FIG. 7

The various components of the environment shown in FIG. 1 and the stepsperformed will now be described. FIGS. 7 to 10 describe virtual mapserver 107, FIGS. 11 to 23 describe the creation of a virtual map, andFIGS. 24 to 27 describe the viewing of a virtual map.

A block diagram of virtual map server 107 is shown in FIG. 7. It is aserver farm comprising three servers 701, 701 and 703. Any suitablenumber of servers can be used, including a single server. In thisexample, server 701 includes a processor which in this example is CPU704, processor memory provided by RAM 705, a data storage medium whichin this example is SSD 706, and an interface provided by modem 707. Anysuitable server architecture may be used. Using current technology, awired connection is necessary in order to serve the data over theinternet 106 but a wireless connection could be used as an alternative.

FIG. 8

Steps carried out by the server 107 to create, store and supply virtualmaps are shown in FIG. 8. At step 801 the server is switched on and atstep 802 a question is asked as to whether the necessary instructionsare already installed. If this question is answered in the negative thenat step 803, a further question is asked as to whether the instructionsshould be installed from the internet. If this question is also answeredin the negative, then at step 804 instructions are installed from aportable storage medium, which could be CD-ROM 805 or some other storagesuch as a flash drive. If the question asked at step 803 is answered inthe affirmative, then at step 805 the instructions are installed from aremote location connected via the internet 106.

Following this, and if the question asked at step 802 is answered in theaffirmative, to the effect that the instructions are already installed,the server instructions are run at step 806. The server is typicallyleft running until maintenance is required, at which point it isswitched off at step 807.

FIG. 9

The contents of memory 705, while the server 107 is running instructionsat step 806, are shown in FIG. 9. At level 901 is the basic operatingsystem, with utilities present at 902 and the server instructions at903. The server may operate an application with a modular structure, sothat it has the ability to work in conjunction with additionalserver-side instructions, and also to serve client-side instructions tothe devices 101, 102, 103, 107 and 108.

Within virtual map editing instructions 904, there are instructions forthe server-side editing application 905 that runs on the server 107,along with instructions for the client-side editing application 906 thatare supplied to a requesting client. Similarly, contained within thevirtual map viewer instructions 907 is the server-side viewerapplication 908 and the client-side viewer application 909. In bothcases, the server-side application operates in a dialogue with theclient-side application, so that the user or customer can performactions seamlessly.

Stored on server memory 705 is virtual map data 910. This includes thevirtual map definitions 911, comprising image and connection data, andthe equirectangular images 912. A virtual map is rendered by displayinga number of images 912 in accordance with data from definitions 911. Allthe map data currently being accessed by users or customers is stored inmap data 910. When a user is editing a virtual map, the data beingcreated on the user's computing device is regularly synchronised withthat stored at 910, so that a user can interrupt a session and pick itup later, possibly on a different device.

Additionally, all map data, including that not currently accessed, isstored within main memory 706. At 913, there is other virtual map dataand at 914 there is other data used by the server.

FIG. 10

Procedures 806, at which the server carries out its instructions, areshown in FIG. 10. The server 107 mainly responds to user requests toedit a virtual map and customer requests to supply a virtual map. Thus,at step 1001 a request is received. This may be received directly or, ifserver 701 is part of a server farm, it may be received via a trafficmanager or other routing system.

At step 1002 a question is asked as to whether the request is from auser wishing to create or edit a virtual map. If this question isanswered in the affirmative, a further question is asked at step 1003 asto whether the user needs to download the instructions for theclient-side editing application 906. If this question is answered in theaffirmative, virtual map editor instructions 906 are transmitted to therequesting device at step 1004. At this point, and if the question askedat step 1003 is answered in the negative, to the effect that the userdoes not require instructions, a session is created to perform datatransfer to or from the requesting device. The user's side of thesesteps will be described further with reference to FIG. 12.

At step 1006 a question is asked as to whether the incoming request isfor supply of a virtual map, and if this question is answered in theaffirmative, a question is asked at step 1007 as to whether the customerneeds to download the instructions for the client-side viewerapplication 909. If this question is answered in the affirmative, thevirtual map viewer instructions are transmitted to the requesting deviceat step 1008. At this point, and if the question asked at step 1007 isanswered in the negative, to the effect that the customer does notrequire instructions, a session is opened at step 1009 to perform datatransfer to and from the requesting device. The customer's side of thesesteps will be described further with reference to FIG. 25.

If the incoming request is neither for the editing or the supply of avirtual map, other server actions may be performed at step 1010. Theseinclude HTML and CSS serving, user account login, session clean-up, andso on. Thus, this loop continues until the server processes areinterrupted by an operation such as the powering down of the server.

FIG. 11

An example of laptop computer 108, used to create or edit a virtual map,is shown in FIG. 11; however, any computing device that has a processor,a memory, and a display device could be used.

The laptop computer 108 includes a central processing unit (CPU) 1101and processor random access memory (RAM) 1102. A data storage medium isprovided by a solid-state drive (SSD) 1103. A graphics processing unit1104 provides visual output data to a display 1105, while a keyboard1106 and a trackpad 1107 provide manual input. A universal serial bus(USB) interface 1108 provides input/output from USB devices, such as thecamera 501, while a Wi-Fi interface 1109 facilitates communication withnetworks, such as the internet 1106.

FIG. 12

Steps carried out by processor 1101, to facilitate the creation orediting of a virtual map, are detailed in FIG. 12. At step 1201 thelaptop computer is switched on and boots up. At step 1202 a web browserapplication is run, and at step 1203 it browses the website of thevirtual map server 107. In this embodiment, the virtual map editor isprovided as an application that runs within a web browser. In otherembodiments it could be a standalone application loaded onto thecomputing device, either via the internet 106 or from a portable storagedevice. However, providing the functionality using the web browserallows the virtual map editor to be used seamlessly as part of theuser's browsing experience, without the user having to install anadditional application.

At step 1204, while browsing the website of the server 107, the userselects the virtual map editor and at step 1205, the processor 1101sends a request for the instructions for the editor from the server,which is processed by the server at step 1004.

At step 1206 the instructions are received from the server and theprocessor loads them in the browser environment. At step 1207, theeditor instructions are executed in response to user input received viakeyboard 1106 and trackpad 1107. This will be further described withrespect to FIGS. 14 to 23.

Virtual map data is regularly synchronised with server 107 during step1207 but at step 1208, all data may be saved in response to userselection of a SAVE function.

Following user selection of a PUBLISH function, the processor 1101requests and receives a URL from server 107, which indicates thelocation at which the virtual map may be viewed. HTML code to embed thevirtual map within a website is also supplied.

At this point, the user may do further work on this or another virtualmap but for the purposes of this illustration, the laptop computer isswitched off at step 1210.

In an embodiment, the virtual map server 107 hosts both the facility tocreate a virtual map and the created maps themselves. However, in otherembodiments a full application for creating a virtual map could bedownloaded as an application to a computing device, such that thecreation of the map would not involve a server. Furthermore, a virtualmap could be hosted by another server, or could be viewed on thecomputing device that created it, or could even be served from the webserver, such as web server 101, that hosts the relevant website. Thecloud environment described herein is only one example of suitablenetwork architecture that can be used.

FIG. 13

A diagram of the contents of RAM 1102, while the laptop computer 108 isexecuting the editing application, is shown in FIG. 13. At a first levelis the operating system 1301, followed by browser instructions 1302.Virtual map editor instructions 1303 are the instructions 906 suppliedby the server 107 to the laptop computer 108 at step 1004, and receivedat step 1206. Editor instructions include image upload process 1309 andimage link process 1310, that enable a user to create the virtual map,and viewer instructions 1311. Instructions 1310 enable a user to viewthe created map and enhance it if required, and will be further detailedwith reference to FIG. 28.

At location 1304, virtual map data includes virtual map definition 1305,images 1306, floor plans 1307, and floor plan data 1308. During asession, these are synchronised with the virtual map definitions 911 andimages 912 in the RAM 705 of the server 107. Floor plans 1307 may beuploaded by the user, and/or may be created using floor plan data 1308as will be described later.

Other data used by the laptop 108 during operation is at location 1309.

FIG. 14

An interface 1401 displayed to the user of laptop 108 on display 1105,is illustrated in FIG. 14, while the editing application is being usedat step 1207. The interface 1401 includes an editing area 1402 and atoolbar 1403.

While creating a virtual map, a user uploads images taken by thespherical camera 501, and circular projections of these are displayedwithin the editing area 1402. In this example, three images have beenuploaded, and are displayed as projections 1404, 1405 and 1406.Projection 1406 is shown with a containing square 1407 which includes arotation point 1408. When a projection is selected, this containingsquare appears; enabling the user to move, size and rotate theprojection in space identified within the editing area 1402.

Projections 1404, 1405 and 1406 are joined by connecting lines 1409 and1410. These are drawn by the user to link points in the images thatconnect to each other. Thus, for example, line 1409 connects door 1411in projection 1404 to door 1412 in projection 1406: these are in factthe same door viewed from different rooms. Similarly, line 1410 joinsthe stairwell 1413 in projection 1404 to the hatch 1414 in projection1405, which are different views of the same linking staircase. Asdescribed with reference to FIG. 3, linking lines are drawn between anytwo points in different images that are different views of the sameconnection point. A connection point may be a doorway, a stairwell, orsimply a point in space (unless a floor plan is to be created from theimages).

The toolbar 1403 contains buttons that, when selected, run functionswithin the editing application 1303. The functions triggered by uploadbutton 1415 and link button 1416 will be described in detail withreference to FIG. 16 and FIG. 22 respectively. Selecting the SAVE button1417 triggers the saving of data at step 1208, and the selection ofPUBLISH button 1418 triggers the publication of the virtual map at step1209. Therefore, the interface 1401 provides the user with an intuitiveway of creating links between images representing physical space.

It is not necessary for the virtual map to include every part of thespace being toured. For example, if two rooms are connected by anunphotographed corridor, then images of the two rooms at either end ofthe corridor could still be linked, even though the rooms are notphysically adjacent. However, if a floor plan is to be created, thenthis is not recommended as it will make it difficult to automaticallyorientate rooms by their connection points, as will be described withreference to FIG. 47.

FIG. 15

The example shown in the interface of FIG. 14 only contains three imagesand two linking lines. However, a typical virtual map would include manyimages and many connection points. A layout that could result from auser creating a virtual map of the house shown in FIG. 2, is illustratedin FIG. 15. Each circle in FIG. 15 represents a projection of an imageof the room indicated, and each line represents a connection point shownby a cross on FIG. 2. A circle 1501 represents an image of the upstairslanding. Thus, a user can orientate the projections so that theconnection points in each are adjacent. This means that the user can, ifthey wish, arrange the layout of the projections to resemble the plan ofthe building, in order to assist in the creation process.

In other embodiments, the interface could display the originalequirectangular images or some other projection, rather than thecircular projections described herein. The user could still defineconnections between points in the images, and the virtual map thusproduced would be indistinguishable from that produced using theinterface of FIG. 14. However, use of the circular projections enablesthe uncluttered, intuitive layout that is possible as shown in FIG. 15.

FIG. 16

Details of upload process 1309 are shown in FIG. 16. The upload button1415 is selected by a user at step 1601. At step 1602, the user isinvited, via a dialog box or similar, to identity the location of theunprocessed equirectangular image data to be uploaded and at step 1602,the image data is uploaded from the identified location and stored inRAM 1102 at the image store 1306. The image may be uploaded directlyfrom the camera 501, connected via USB interface 1108, or it may havebeen previously stored in data storage medium 1103 or at a networkedlocation.

At step 1604, the equirectangular image data is converted to a circularprojection, described further with reference to FIG. 18, and at step1605, this projection is displayed within the interface 1401. At step1606 an indication that this image has been added to the map is storedin virtual map definition 1305.

If during the editing process, the user decides to delete one of theprojections, the indication of the image is deleted from the virtual mapdefinition 1305 and the original image is removed from image store 1306.

FIG. 17

The conversion of an equirectangular image 601 to a circular projection1701 is shown in FIG. 17. Image 601 includes a considerable amount ofspace dedicated to the floor and ceiling. Since connections aregenerally described between points visible at wall level, it isunnecessary to display either the floor or the ceiling in the circularprojection 1701. Therefore, in this embodiment, the ceiling is removedfrom the image before it is processed into a circular projection, whilethe floor is omitted during the process in order to leave a blank circle1702, central to circular projection 1701. The floor and the ceilingcould be retained in other embodiments of the invention. However, theirremoval creates a circular projection that is smaller and easier to use.

Returning to image 601, a first line 1703 indicates the cut-off for theceiling and a second line 1704 indicates the cut-off for the floor. Inan equirectangular projection produced by a spherical camera, both theceiling and the floor take up approximately twenty-five per-cent of theheight of the image each. Consequently, the top twenty-five per-cent ofthe image is removed to produce a cropped image 1705. This cropped imagenow has a useable area of the top sixty-seven per-cent, while the bottomthird is shown as blank, because it will not be transferred to thecircular projection.

Thus, the circular projection 1701 is a projection of the middle half ofimage 601 into an annulus (bagel or doughnut shape) wrapped around acentral point.

FIG. 18

Procedures 1604 of the upload process are detailed in FIG. 18, at whichthe uploaded equirectangular image, for example image 601, is convertedto a circular projection, for example projection 1701. Further detailsof this process are described with reference to FIGS. 19 to 21.

At step 1801 a containing square is instantiated. This square has astandard height and width of a set number of pixels. It is processedpixel by pixel from the top left, therefore at step 1802, the first rowin the square, r, is selected, and at step 1803 the first column in thatrow, c, is selected. This leads to the selection of a pixel point thatcan be considered as point (c, r). At step 1804 the distance D from thecentre of the square to this point is calculated and at step 1805 aquestion is asked as to whether this distance measurement indicates thatthe point in the square is within the annulus created by predeterminedthresholds of the circular projection. In an embodiment, the outercircle of the annulus has a diameter similar to the width of the squareand the inner circle has a diameter that is one third of that of theouter circle, with both circles being centred with respect to thecontaining square; this is described with reference to FIG. 19.

If the question asked at step 1806 is answered in the affirmative, tothe effect that this point is within the annulus, then at step 1806, thesource pixel in the original equirectangular image that corresponds tothis point is identified. At step 1807 the alpha channel of the sourcepixel is modified if necessary before the pixel information is saved inthe position of point (c, r).

If the question asked at step 1805 is answered in the negative, to theeffect that the selected point is not within the annulus, then steps1806 to 1808 are skipped and this point is left blank. This creates awhite space within the containing square, outside the outer circle andinside the inner circle.

A question is then asked at step 1809 as to whether there is anothercolumn in the selected row. If this question is answered in theaffirmative, control is returned to step 1803 and the next column isselected. If it is answered in the negative, to the effect that this rowhas been processed, then at step 1810, a question is asked as to whetherthere is another row of pixels in the containing square. If thisquestion is answered in the affirmative, control is returned to step1802 and the next row is selected. If it is answered in the negative,every point in the containing square has been processed and step 1604 iscomplete.

Thus, during step 1604, each point in the containing square isconsidered, such that it is either left blank or pixel data from theoriginal equirectangular image is imported.

The process described herein considers the square on a pixel by pixelbasis, but in alternative embodiments, if lower resolution is anacceptable compromise in exchange for faster processing speed, theprocess may, for example, average out pixel information.

The containing square thus produced is the image displayed to the userwithin interface 1401. When the user selects one of the circularprojections, the containing square is displayed as square 1407, with arotation point 1408.

FIG. 19

FIGS. 19 and 20 show the formulae used in the steps carried out in FIG.18. FIG. 19 shows how the distance from the centre of a containingsquare 1901 to a selected point (c, r) is used to determine whether thepoint is filled in, with pixel information from the originalequirectangular image, or is left blank.

Square 1901 is instantiated with a standard width and height of 2R,where R is a predetermined number of pixels. The outer circle 1802 has aradius of R. The inner circle 1903 has a radius of R/3; thus, referringback to FIG. 17, it can be seen that the space between the two circles1902 and 1903 will contain the top two thirds of image 1705, whereas theinner circle, rather than continuing the lower third of the image, willbe left blank.

If the top left point of square 1901 is considered to be at coordinates(0, 0), the central point is at coordinates (R, R). The distance Dbetween the central point and any given point (c, r) is therefore givenby formula 1904:D=√{square root over ((c−R)²+(r−R)²)}

Thus, if the distance D is less than R/3 or greater than R, as shown at1905, the point is outside the annulus, in which case the question askedat step 1805 is answered in the negative and the point is left blank. Inthe example shown in FIG. 19, the point (c₁, r₁) is outside outer circle1902 and will therefore have a value of D greater than R, meaning thatit will be left blank. Point (c₂, r₂) is within the annulus andtherefore pixel information from the source image will be imported.Point (c₃, r₃) has a value of D less than R/3 and is therefore insideinner circle 1803 and will be left blank.

The thresholds used in this method are dependent upon the removal of thelower one-third from the cropped rectangular image. If a smaller orgreater amount is to be removed, then the threshold to decide whether apoint inside the containing square is inside the annulus would becorrespondingly different. In addition, this method crops the toptwenty-five per-cent from the original image before generating thecircular projection, but a method that removed it by use of a threshold,as the lower part is removed, would also work. In addition, in otherembodiments the entire original image could be used to create theprojection, particularly if it was the case that links between the floorand ceiling of adjacent rooms were required, for example to link anattic space by way of a roof hatch.

FIG. 20

FIG. 20 shows the formula used during step 1806 to identify the sourcepixel in the originating image. In this illustration, the originalequirectangular image is shown at 2001, while the circular projectioncreated from it is shown at 2002. In order to determine a positionwithin the source image 2001, a point (c, r) in the containing square isconverted to a point (x, y) in the source image 2001. This source imagehas already had the top twenty-five per-cent removed and thereforecorresponds to image 1705. Using the image processing convention thatthe point (0, 0) is at the top left, the x coordinate therefore variesfrom 0 to W, where W is the width of the data in pixels, and y is thecoordinate that varies from H at the bottom to 0 at the top of theimage, where His the height of the image in pixels after the toptwenty-five per-cent has been removed. Thus, any point with a ycoordinate larger than 2H/3 will fall within the inner circle and willnot be displayed.

In order to determine an x coordinate, the angular position of the point(c, r) around the central point (R, R) is determined using formula 2004:θ=atan 2(R−c,r−R)

The function atan 2 (which is well known and will not be set out herein)is a discontinuous function that returns a value between −π and π, asshown at 2005. This value corresponds to the angular distance, taken inan anti-clockwise direction, between the line 2003 and the point (c, r).Thus, looking at the example of 2002, point (c₂, r₂) returns a value of8 that is smaller than the value returned by point (c₄, r₄).

Line 2003 represents the join between the two edges of image 2001 whenit is projected. Therefore, the formula to convert the variable 8 intoan x coordinate in image data 2001 is given by formula 2006:

$x = {\left( {1 + \frac{\theta}{\pi}} \right)\frac{W}{2}}$

The previously calculated distance D from the point (c, r) to the centreof the circle is used to identify the corresponding y coordinate inimage 2001 using formula 2007:

$y = {H - \frac{HD}{R}}$

Thus, for example, the point (c₂, r₂) in projection 2002 corresponds tothe point (x₂, y₂) in image 2001, and the pixel data from that point isimported to point (c₂, r₂). Similarly, the pixel information from point(x₄, y₄) is imported to point (c₄, r₄) in projection 2002.

The pixel information imported is the RGB (red, green, blue) and alpha(opacity) values for the selected pixel. The alpha value may be modifiedbefore importing, as will be described further with reference to FIG.21. Other embodiments may use other methods of creating a circularprojection from an equirectangular image.

FIG. 21

FIG. 21 details step 1807, at which the alpha channel of the sourcepixel is modified if it is close to the outer or the inner edge of theannulus. The value of alpha for all the pixels in the source image isassumed to be one.

At step 2101 a first distance is calculated to be the distance from theouter circle to the point under consideration. In this embodiment, thisfirst distance is calculated as R−D. At step 2102 a question is asked asto whether this first distance is less than one, and if this question isanswered in the affirmative then, at step 2103, the value for the alphachannel when the source pixel is copied is set to be this firstdistance.

If the question asked at step 2103 is answered in the negative, to theeffect that the first distance is greater than one then, at step 2104, asecond distance is calculated, which is the distance from the pointunder consideration to the circumference of the inner circle. In thisembodiment, this is calculated as D−R/3. At step 2105 a question isasked as to whether this second distance is less than one and if thisquestion is answered in the affirmative then, at step 2106, the valuefor the alpha channel when the source pixel is copied is set to be thissecond distance.

However, if the question asked at step 2105 is also answered in thenegative then, at step 2107, the alpha is left unaltered at one. Inother embodiments, if the alpha channel of all pixels in the sourceimage was not one, then at this step, the alpha value could be eitherset to one or left unaltered.

These steps have the effect of reducing the alpha, and therefore theopacity, of the pixels at the very edges of the annulus. This reducesthe appearance of jagged lines and corners that would otherwise bepresent. In other embodiments, the formulae for calculating the firstand second distances could be different, leading to a wider band ofreduced opacity at each edge.

Thus, at the conclusion of the steps detailed in FIG. 18, a circularprojection of the original equirectangular image data has been generatedand displayed on display 1105. The user can now easily create linksbetween these projections in order to define connections between thespaces photographed in the original images.

FIG. 22

FIG. 22 details image link process 1310, initiated by a user selectionof link button 1416 at step 2201.

At step 2202 a question is asked as to whether at least two circularprojections are displayed, and if this question is answered in thenegative then an error message is displayed at step 2203 and the processis terminated. In this embodiment, it is not possible to create a linkfrom a projection to itself.

However, if the question is answered in the affirmative then at step2204, the user selects a point on each of two projections and, at step2205, a line is displayed between these two points on the display,indicating to the user that the link has been created. At step 2206 theselected points are converted to normalised polar coordinates and storedin virtual map definition 1305.

The polar coordinates stored are with reference to the circularprojection. However, they are easily transformed into pitch and yawvalues that can be used with reference to the equirectangular images inimage store 1306, as will be described further with reference to FIG.27.

FIG. 23

FIG. 23 shows a database structure suitable for storing the virtual mapdata. This is the virtual map definition stored at 911 on the server107, at 1305 on the laptop 108, and at 2605 on the tablet 103, and whichis described with reference to FIG. 26.

The database contains two tables. The first, images table 2301, containsa list of images contained in the map, their locations within the imagestore 912, 1306 or 2606, and their alphanumeric names. The second, linkstable 2302, contains, in each record, references to two images fromtable 2301. For each image, polar co-ordinates are given as an angle anda radius.

Thus, each line in table 2302 defines a link between a point in each oftwo different images. Each link is displayed as a connection point inthe virtual map viewer.

Within virtual map definitions 911 in the server memory 705, a pluralityof database structures such as this are stored, one for each virtualmap. However, each requesting device only holds one such structure inmemory, for the map currently being edited or viewed.

The structure shown in FIG. 23 is an example of position data, thatstores identifications of positions within images, and an indicationthat at least one pair of such positions is connected. The position datacould take other forms in other embodiments.

FIG. 24

The process of a customer viewing a virtual map will now be describedwith reference to FIGS. 24 to 27. FIG. 24 is a diagrammaticrepresentation of tablet 103 that is, in this example, used to view avirtual map. However, as previously indicated, any computing device witha processor, memory and a display can be used.

The processor in the tablet 103 is provided by CPU 2401, and processormemory by RAM 2402. A data storage medium is provided by a flash drive2403. The tablet further includes a graphics processing unit 2404 whichoutputs to a display 2405. A touch panel processor 2406 receives inputfrom display 2405. Interfaces are provided by USB interface 2407 andWi-Fi interface 2408.

FIG. 25

FIG. 25 details steps carried out by processor 2401 in order to allow acustomer to view a virtual map. At step 2501, tablet 103 is switched onand at step 2502 a web browser application is run. Similarly, to theediting application, the viewing application is run within a browser.This enables a customer to view a virtual map without downloading aspecific application to their computing device. However, in otherembodiments a specific application could be used.

At step 2503, the customer browses to a website that includes anembedded virtual map, and at step 2504 the user selects a virtual map tobe viewed. This provides the browser application with a URL from whichthe virtual map may be loaded. This URL is located on virtual map server107; however, as previously described, the virtual map could be hostedin any location.

When the processor requests the virtual map from map server 107, theserver processes this request at step 1008 and transmits instructions909 for the client-side viewer application. These are received atprocessor 2401, stored in memory, and loaded within the browserenvironment at step 2506.

The server then provides the virtual map at step 1009. At step 2507, thevirtual map is displayed within the browser, and at step 2508, theapplication responds to customer input to navigate the map. When theuser has finished, at step 2509, the tablet is switched off.

FIG. 26

FIG. 26 shows the contents of memory 2402 while processor 2401 isrunning the virtual map viewing application at steps 2506 and 2507.

At level 2601 is the operating system 2601, along with browserinstructions 2602. At 2603 are virtual map viewer instructions. Theseare the instructions received from virtual map server 107 at step 2506.Virtual map data 2604 includes the virtual map definition 2605 andimages 2606 received from server 107. Other data 2607 facilitates theoperation of tablet 103.

FIG. 27

FIG. 27 details step 2508, at which processor 2401 responds to customerinput while displaying the virtual map using viewer instructions 2603.

In this embodiment, viewer instructions 2603 are provided in HTML5,allowing them to be run transparently in a web browser application.Other platforms could be the widely available Flash viewer orJavaScript, but any suitable platform may be used.

The viewer loads a selected equirectangular image and displays it withina browser in a panoramic or spherical format that can be easilynavigated by the customer. In addition, the viewer interrogates virtualmap definition 2605 to determine whether the image contains anyconnection points that should be displayed. If so, the position of theconnection point is obtained along with the name of the image that itleads to, and these are displayed within the image at the correct point.

Points within a panoramic viewer are usually described in sphericalcoordinates, including pitch and yaw, and the normalised polarco-ordinates stored in map definition 2605 can be converted as follows:pitch=−90+(135*radius)yaw=360*angle

Thus, a display projection of the image is rendered, which includes adisplay element, which in this example is a connection point. Thisdisplay element corresponds to a position from the position data held inlinks table 2302.

After displaying the first image at step 2507, customer input isreceived at step 2701. This may be received via any suitable method ofproviding input, such as a touchscreen or touch pad, mouse or keyboardinput, selection of an on-screen button, tilting a mobile device, etc.In this example, the customer uses the touchscreen of tablet 103 toprovide input.

At step 2702 a question is asked as to whether the input is a requestfor movement of the image, and if this question is answered in theaffirmative then at step 2703 the request is responded to by alteringthe view of the displayed image at step 2703.

However, if this question is answered in the negative, then at step2704, a further question is asked as to whether the input is a selectionof a connection point, which in this embodiment is displayed as alabelled target, for example as shown in FIG. 3.

If this question is answered in the affirmative then at step 2705virtual map definition 2605 is interrogated to identify thecorresponding connection point that is linked with the selectedconnection point. Referring back to FIG. 23, this identifies a furtherimage file and polar coordinates within that file. Thus at 2706 theidentified image is loaded, and at step 2707 the image is orientatedsuch that it is from the point of view of the corresponding connectionpoint. This means that the point in the displayed panoramic image thatis 180° from the connection point is substantially in the middle of thefield of view, as displayed in the viewer. Thus, the connection point is“behind” the viewer in the displayed panoramic image, and the customerwould need to turn the image by exactly 180° to bring the connectionpoint to the centre of the field of view.

At step 2708, the orientated image is displayed. Thus, the customer hasthe virtual experience of walking through the connection point with nochange of orientation.

Thus, a display projection of the next image is rendered, which alsoincludes a display element. This display element corresponds to aposition from the position data held in links table 2302.

If the question asked at step 2704 is answered in the negative then, atstep 2709, other input is responded to. This could be, for example, arequest to change or zoom the view, to enter a virtual reality modewhere a split screen is displayed suitable for a virtual reality viewer,to display a floorplan, to enter or exit full screen mode, and so on.

FIG. 28

The foregoing description describes how images in a spherical format maybe connected together in order to create a virtual map of a building.FIGS. 28 to 48 describe methods of enhancing such images in sphericalformat in order to provide a better user experience, and/or to extractfurther data from the images. These are collectively considered to beenhancements to the images. In order to carry out these enhancements, itis not necessary, except in certain situations that will be detailed,for the user to have created a virtual map as previously described.However, the enhancements are described within the context of theproduction of a virtual map.

Enhancements to the images in spherical format are carried out, in thisembodiment, after the user has created a virtual map at step 406.However, they could be carried out at any time after a user has uploadedan image at step 402.

The enhancements are carried out by viewer instructions 1311 on thememory 1102 of laptop 108. These instructions comprise instructionsidentical to instructions 2603, but include further processes. Theseinclude camera pose compensation process 2801, floor plan creationprocess 2802, and cursor movement process 2803. These will now bedescribed with reference to the remaining Figures.

In other embodiments, some or all of the enhancement processes could berun on virtual map server 107, particularly if they are to be appliedautomatically without user input.

FIG. 29

FIG. 29 shows a user interface 2901 displayed on display 1105 of laptop108. Interface 2901 includes a viewing/editing area 2902, in which canbe seen an image 2908 displayed in a spherical format, and a tool bar2903.

Tool bar 2903 is similar to tool bar 302, displayed to a user viewing avirtual map. However, viewer instructions 1311, being part of thevirtual map editor instructions 1303, contain additional functionalityto enhance image 2908. In this embodiment, the functionality is providedby camera pose compensation button 2904 and floor plan creation button2905, which initiate processes 2801 and 2802 respectively. Cursormovement process 2803 is initiated during floor plan creation process2802, and causes the display of a cursor that replaces the usual cursorof the spherical viewer, as will be described further with reference toFIG. 39.

In the example shown in FIG. 29, the user has created connection points2906 and 2907 to other rooms in the virtual map. However, both processes2801 and 2802 may be used to enhance a single image, and therefore touse them it is not necessary that images of additional rooms have beenuploaded nor that connection points have been established.

As previously described, image 2908 shown in area 2902 is in a sphericalformat, with the image being wrapped around the inside of a notionalsphere. Therefore, as the user moves a cursor around the interface,spherical coordinates representing pitch and yaw are provided to viewerinstructions 1311. These are used to allow the user to move the viewpoint around, providing a 360° view of the room, and to allow the userto zoom in and out while maintaining the spherical format. However, theimage contains no data or coordinates relating to whether particularareas of the image relate to a floor, a wall, a ceiling, etc. Each pixelis merely a point on a sphere and no further information is available.Further, each pixel is considered to be the same distance away from thecentre of the notional sphere, and therefore no information is availablerelating to distances within the physical space shown in image 2908.

The invention as described herein provides a method of converting thespherical coordinates of a cursor point to two-dimensional Cartesiancoordinates on a specific plane within the sphere. Thus, for example, ifthe floor is a plane of the sphere, then spherical coordinates can beconverted to two-dimensional Cartesian coordinates on the floor, thusproviding a floor plan of the room. This will be described further withreference to FIGS. 38 to 48.

In addition, if the camera was not perfectly vertical when image 2908was taken, the spherical format will not be perfectly aligned with thephysical space that was imaged. By transforming spherical coordinates ofparticular positions within the image to three-dimensional Cartesiancoordinates, it is possible to estimate the orientation of the room withrespect to a perfect vertical, and therefore either adjust the image sothat it appears aligned on the screen, or compensate for thisorientation when performing other enhancements. This will be describedfurther with reference to FIGS. 33 to 37.

FIG. 30

The method of transforming spherical coordinates into two-dimensionalCartesian coordinates on a plane is illustrated in FIG. 30. Viewerinstructions 1311 create the spherical format of an equirectangularimage or other image by wrapping it around the inside of a notionalsphere 3001, centred on the position 3002 of the camera when thephotograph was obtained. Every point in the spherical format is definedby spherical coordinates of pitch and yaw. The sphere can be consideredto have an internal three-dimensional Cartesian coordinate system,including an x-axis 3003, a y-axis 3004 and a z-axis 3005. The sphericalcoordinates of pitch and yaw are measured relative to the plane ofsphere 3001 where z is equal to zero, i.e. the plane containing the x-and y-axes. Yaw is measured parallel to this plane from the x-axis, andpitch is measured from this plane.

Thus, a value of 0° for yaw represents the viewer looking straight aheaddown the x-axis. −90° indicates looking left, 90° indicates lookingright, and either 180° or −180° (depending on the configuration of theviewer instructions) indicates looking backwards. For pitch, a value of0° indicates looking level with the camera, −90° indicates lookingstraight down to the bottom of the sphere, and 90° indicates lookingstraight up to the top of the sphere. Values of pitch outside the rangeof −90° to 90° are not used, as in a typical 360° viewer it is notgenerally necessary to turn the image upside down.

Therefore, point 3006 on the surface of sphere 3001 has a yaw angle 3007and pitch angle 3008. In the example shown in FIG. 31, both the yaw andthe pitch are negative, indicating that the viewer is looking to theleft and down. This pitch and yaw value uniquely identify the point3006, and therefore a point within image 2908. However, they provide noinformation as to whether the point in the image represents a positionon the floor, the ceiling, or a wall. The viewer simply rotates a spherearound a central point with no information regarding the actual physicalspace represented by that image.

It is possible to discover the distance from the floor to the camera, atthe time at which the image was obtained by the camera. It is alsoreasonable to assume that the floor is horizontal. Therefore, ahorizontal plane 3010 can be constructed within sphere 3001, that is ata distance 3011 from the centre 3002 of the sphere. Distance 3011represents the height of the camera from the floor. In this situation,horizontal means parallel to both the x-axis and the y-axis. Bydiscovering the point 3012 at which the line from the centre 3002 of thesphere to point 3006 intercepts plane 3010, a position intwo-dimensional Cartesian coordinates on plane 3010 can be obtained forpoint 3012.

These two-dimensional Cartesian coordinates represent an actual locationon the floor in the physical space imaged, for a point in image 2908having the specified spherical coordinates. For example, returning toFIG. 29, if the cursor is placed on an area of image 2908 thatrepresents the floor, the Cartesian coordinates of that location on thefloor will be calculated. If a position on the wall is selected, thenthe Cartesian coordinates will identify the location on the floor thatwould exist if the floor were extended beyond the wall. Therefore, byidentifying discontinuities in the image that indicate the intersectionof a wall and the floor, it is possible to calculate Cartesiancoordinates for these discontinuities, and thereby produce a floorplanof the room. These discontinuities may be identified either by a userselecting corners or edges, or by an automatic process that tracesedges.

In addition, identifying Cartesian coordinates of locations can be usedto provide an orientation of image 2908. Walls, floors and ceilings inrooms are generally at right angles to each other, and therefore if aplurality of intersections between walls are identified and theirCartesian coordinates calculated, it is possible to identify anorientation that sets the expected right angles to be as near to 90° aspossible.

In addition to a plane corresponding to the floor, a plane correspondingto the ceiling can be constructed if the height from the floor to theceiling is known. In this case, the two-dimensional Cartesiancoordinates would be with respect to the ceiling plane. This can be usedinstead of identifying locations on the floor, or in addition. If bothplanes are used, then three-dimensional Cartesian coordinates can beidentified, and used to give greater accuracy to either method describedabove.

In addition, other methods of enhancing an image in spherical format mayuse this transformation from spherical coordinates to Cartesiancoordinates.

FIG. 31

The equations used to calculate a point location in two-dimensionalCartesian coordinates are shown in FIG. 31, along with a diagramindicating their derivation. Plane 3010 is shown as if looking at itfrom the top of sphere 3001, i.e. backwards down the z-axis 3005. Thex-axis 3003 and the y-axis 3004 are therefore shown at right angles toeach other. The two-dimensional Cartesian coordinates of point 3012 withrespect to the x-axis and the y-axis can be calculated if the distance3101 from the centre 3102 of the plane to point 3012 is known, using yawangle 3007. The equations to calculate this are shown at 3110 and 3111as follows: the value on the x-axis is distance 3101 multiplied by thecosine of the yaw angle 3007; the value on the y-axis is distance 3101multiplied by the sign of the opposite of yaw angle 3007, which is tosay the yaw angle multiplied by minus 1.x=D*cos(YAW)y=D*sin(−YAW)

Distance 3101 can be found using the pitch angle 3008 and the height3011 of the camera from the floor, which is considered to be thedistance from centre 3002 of sphere 3001 to the centre of plane 3010.Returning to FIG. 30, it can be seen that a right-angled triangle ismade between the centre 3002 of the sphere, point 3012, and acorresponding point on the plane defined by the x-axis and the y-axis,with one of the angles of this triangle being pitch angle 3008.

The equation to calculate distance 3101 is shown at equation 3112, asfollows: the height 3011 divided by the tangent of the absolute value ofpitch angle 3008. Alternatively, it can be written as height 3011multiplied by the tangent of the other angle in the triangle, i.e. pitchangle 3008 subtracted from 90 degrees.D=H/tan(abs(PITCH))D=H*tan(abs(90°−PITCH)

The second calculation may be preferred in a programming environment asit does not risk a division by 0.

Therefore, after the above calculations have taken place, a value on thex-axis and a value on the y-axis have been obtained, providingtwo-dimensional Cartesian coordinates corresponding to the location onthe floor of the point in the image having spherical coordinatesidentified by point 3006.

Other planes can be used besides the floor. For example the ceiling maybe used, as previously described, if the height of the ceiling from thefloor is known. In this case, the height used in equation 3112 would bethe height of the camera subtracted from the height of the ceiling. Thefact that the ceiling is to be considered is identified by a positivevalue for the pitch.

Thus, there is provided a method of identifying a location on a surfacerepresented in a spherical image, where the surface was at a knowndistance from the camera and had a known orientation when the image wasobtained. Any point in the spherical format is defined by sphericalcoordinates consisting of a yaw angle and a pitch angle defining a linefrom the centre of the sphere. A plane may be identified having theorientation of the surface that is at the known distance from thecamera, i.e. the centre of the sphere, and a location in a Cartesiancoordinate system can be calculated on the plane at the point where itintersects with the line, thereby identifying the location of the pointon the surface.

If the camera is outside, then the ground would be the surface underconsideration. However, it is not necessary that the surface behorizontal. This method could be used to identify points on any surfacehaving a known orientation and being at a known distance from the camerawhen the image was obtained. The Cartesian coordinates obtained would bewith respect to the orientation of that plane, which could then betransformed if required into the Cartesian coordinates system of thesphere. For example, a horizontal plane could be used to represent awall, if the distance from the wall to the camera were known. Thecoordinates thus obtained from identifying corners of the wall could beused to identify the height of the room. The plane not need even behorizontal or vertical but could be at any rotation to the coordinatesystem of the sphere. For example, if the ground were at a known slope.

Two uses of this method will now be described. First, a method oforientating an image in a spherical format is described with referenceto FIGS. 32 to 37. Next, a method of obtaining a floor plan, either of asingle room or of an entire building, is described with reference toFIGS. 38 to 48. Other uses for this method are envisaged.

FIG. 32

The image 2908 in spherical format shown in FIG. 29 is a projection ofan image that was taken by a camera. The camera may not have been on acompletely vertical tripod. In fact, it is very likely that such was thecase, as floors are rarely perfectly flat and tripods are rarelyperfectly vertical. The degree to which the camera deviates fromvertical is known as the camera pose, and it consists of a pose-pitchangle and a pose-roll angle. If the camera is looking straight forward,then the pose-roll indicates the degree to which it is leaning to theleft or right, and the pose-pitch indicates the degree to which it isleaning forwards or backwards.

Many spherical cameras have internal gyroscopes which provide thepose-roll and pose-pitch angles, and if an image was taken with one ofthese cameras, then these angles are stored with the image metadata.However, many cameras do not include a gyroscope. The method describedherein therefore provides an estimation of these of the camera pose.

FIG. 32 is an illustration of camera 501 mounted on tripod 502, in theprocess of obtaining image 2908. A dotted line 3201 indicates an exactlyvertical position, and it can be seen that camera 501 is deviating fromthat. For the purposes of viewing the image produced, it is unlikelythat a user would notice the small deviation. However, in order toprovide an improved experience, the creator of the virtual map may wishto ensure that all of the images are orientated to provide an exactreplica of the physical space imaged.

In addition, if it is intended to use the method described with respectto FIG. 31 to create a floor plan or identify locations on surfaces forsome other reason, then any slight deviation in the orientation of theimage can cause errors, particularly if two planes are to be consideredsuch as the floor and the ceiling. Therefore, if the camera pose can becompensated for, this will produce more accurate results.

FIG. 33

Camera pose compensation process 2801 is initiated by a user pressingbutton 2904 on tool bar 2903 at step 3301, which is now marked asselected. At step 3302 a question is asked as to whether the camera thattook the image had a gyroscope. This will be answered in the positive ifthe metadata for the image comprises camera pose angles, and if so theseare retrieved at step 3303.

However, if there are not camera pose angles in the metadata, then atstep 3304 the angles are estimated by identifying right angles in theimage.

In either case, the question is then asked as to whether the imageshould be rotated at step 3305. This is a matter of user preference. Ifthe question is answered in the affirmative, then at step 3306 the image2908 is adjusted by transforming all its pixels by the pitch-pose andpitch-roll angles indicated.

Alternatively, if the question asked at step 3305 is answered in thenegative, then the pitch-pose and pitch-roll values are stored in theimage metadata and used to modify calculations whenever required, aswill be described with reference to FIGS. 37 and 44.

Camera pose compensation process 2801 could comprise part of the uploadprocess 1309, taking place whenever a user uploads an image rather thanafter a virtual map has been created. The process is also independent ofthe use made of the images, and could be used to enhance any 360°viewer, not just the virtual map viewer described herein.

FIG. 34

In order to estimate the pose-pitch and the pose-roll angles of camera501 when it obtained image 2908, the method described herein exploitsthe fact that most angles in a room are right angles. Generally, a wallis at right angles to both the floor and the ceiling, and usually mostwalls are at right angles to each other. For a user viewing thespherical image of the room, it is reasonably simple to identify rightangles. However, they do not appear as right angles in image 2908,either in its original panoramic format or in the spherical format ofthe viewer. Therefore, an automated computer system would not be able toidentify right angles using any known process.

To estimate the camera pose, the user draws edges in the image that formright angles with one another. An example of this is shown in FIG. 34,where the user has drawn rectangles, one of each of three walls, whereit would be expected that the intersection of each pair of lines wouldbe a right angle. For example, the user draws line 3407 indicating theintersection between the floor and a wall, and a line 3408 indicatingthe intersection between the wall and another wall. The user expectsthat the angle 3409 between these lines would be a right angle. 3407 isdrawn between end points 3405 and 3406,

Each of these lines is defined by two end points, each having sphericalcoordinates. For example, line 3407 is defined by endpoints 3405 and3406. These spherical coordinates only identify points on the sphere,and therefore cannot be used to estimate the camera pose. However, bytransforming each set of spherical coordinates into a set of Cartesiancoordinates, the angles between each pair of intersecting lines can beestimated.

In order for the spherical coordinates to be transformed intotwo-dimensional Cartesian coordinates, it is necessary to know thedistance from the floor to the camera. In addition, to transform theminto three-dimensional Cartesian coordinates, it is also necessary toknow the distance from the floor to the ceiling. Therefore, input boxes3401 are provided within interface 2901 to allow the user to specifythese measurements. In this example, the user is asked for the height3402 of the tripod, the device height 3403, ie the distance 3403 fromthe tripod to the camera lens, and the height 3404 of the ceiling.Alternatively, a database of known devices could be stored and the usercould select the device used. The tripod and camera heights could thenbe retrieved from the database.

If the height of the ceiling is not known, then it is still possible tocarry out the method of estimating the camera pose. However, the userwould be restricted to selecting points on the floor only. This mightmake the estimation less accurate as fewer points would be provided. Inaddition, the corners on the floor may be obscured by skirting boards,furniture etc.

If the height of the camera is not known, then it may be possible toestimate it from items in the room that are expected to be at a knownheight on the wall, for example light switch 3410. This will bedescribed further with reference to FIG. 36.

FIG. 35

FIG. 35 details step 3304, at which the camera pose angles areestimated. At step 3501, the height of the camera is obtained, bysumming the heights of the tripod and the device given at 3402 and 3403,or by any other relevant calculation. At step 3502 the user draws‘perpendicular’ lines in the room. This is by drawing lines along edgesthat the user is aware are at right angles to each other; however, thelines will not appear to be perpendicular in the spherical format, ascan be seen in FIG. 34.

The process then iterates through a pre-determined set of pose-pitch andpose-roll values. In this example, these values are between −15° and15°, and are iterated by 1° each time. However, in order to achievegreater accuracy a small iteration size could be used. In addition, agreater range could be used. However, 15° in any direction is consideredto be the maximum deviation in an image that a user would tolerate.Outside this range, it is likely that the image would be obviouslyskewed and the photographer would retake it.

Therefore at step 3303, the first pair of pitch and roll values isselected. An example of the implementation of this step would be toselect a roll value, then iterate through all possible values for pitch,and repeat for all the roll values, or vice versa. However, for ease ofexplanation in this Figure, this is not detailed.

At step 3504 a first endpoint of a line drawn by a user is selected, forexample endpoint 3405, and at step 3505 the spherical coordinates ofthis endpoint are rotated by the current pair of pitch and roll values.

At step 3506 the rotated spherical coordinates are transformed toCartesian coordinates, using the method described with reference to FIG.31. This will be detailed further with reference to FIG. 36. At step3507 a question is asked as to whether there is another endpoint in theset of points selected by the user, and if this question is answered inthe affirmative control is returned to step 3504.

If all the endpoints have been transformed, then at step 3508 the firstangle between intersecting lines is selected for consideration, forexample angle 3409 between lines 3407 and 3408. Using the Cartesiancoordinates identified for the endpoints at each iteration of step 3506,an angle in Cartesian space can be calculated between the lines. At step3509 the absolute deviation from 90° of that angle is added to a runningtotal, and at step 3510 a question is asked as to whether there isanother angle in the set of lines. If this question is answered in theaffirmative then control is returned to step 3508 and the next angle isselected.

If the question is answered in the negative, to the effect that all theangles have been considered, then the total deviation is stored for thispair of pitch and roll values, and the total is reset to zero ready forthe next pair at step 3511.

Therefore, at step 3512 a question is asked as to whether there isanother pair of pitch and roll values to be considered, and if thisquestion is answered in the affirmative then control is returned to step3503 and the next pair is considered. If the question is answered in thenegative, all possible pairs of pitch and roll values have beenconsidered and a total deviation has been stored for each one.Therefore, at 3513 the pitch and roll pair with the lowest totaldeviation is selected as the best estimate of the camera pose angles.

This method assumes that the angles in the room are right angles.However, if a user was aware that a room was an unusual shape andcontained a number of alternative angles, then it would be possible tospecify this angle and select only the relevant angles in the room. As afurther alternative, the user could specify an angle for every pair ofintersecting lines selected.

In an alternative embodiment, rather than the user selecting the rightangles, the process could automatically select intersections betweenwalls, floor and ceiling by identifying discontinuities of colour ortexture and thereby tracing the edges. In this case, the method could becarried out automatically at server 107 whenever a user uploaded animage, thus providing an automatic reorientation of images.

FIG. 36

FIG. 36 details step 3506 at which the spherical coordinates of arotated end point are transformed to Cartesian coordinates. These stepsare also carried out during floor plan creation process 2802 at step4303, as will be described with reference to that Figure.

At step 3601, a question is asked as to whether the pitch angle of thespherical coordinates is positive. If this question is answered in thenegative, then the viewer is looking downwards, and the Cartesiancoordinates should be calculated with respect to a plane below thecentre of the sphere, which in this case is the floor. Thus, at step3602 the height used in the calculation is set to be the camera heightas obtained at step 3501. In addition the third coordinate, on thez-axis, is set to be 0.

Alternatively, if the pitch is positive, then the viewer is lookingupwards and the location should be calculated with respect to a planeabove the centre of the sphere, in this case the ceiling. The heightused in the calculation is therefore set at step 3603 to be the cameraheight subtracted from the ceiling height. The third coordinate on thez-axis is set to be the ceiling height.

Following the determination of the height value at either step 3602 orstep 3603, then at step 3604 the distance along the plane in question iscalculated by multiplying the height by the tangent of the absolutevalue of the pitch subtracted from 90°. This is the second version ofequation 3112, which is preferred because it uses multiplication ratherthan division.

Once the distance has been calculated, then at step 3605 the coordinateson the x-axis and the y-axis can be calculated using equations 3110 and3111.

Thus, at the conclusion of step 3506, three-dimensional Cartesiancoordinates have been obtained for the end point, rotated by the currentpair of pitch and roll values. If the ceiling height were not known,meaning that only points on the floor were to be selected, then thepoints could be determined in two-dimensions by omitting the value onthe z-axis.

Thus, there is provided a method of identifying the orientation of animage in a spherical format where a surface represented in the image hada known orientation and was at a known distance from the camera when theimage was obtained. A plurality of lines are obtained in the sphericalformat, each line defined by two end points in spherical coordinates,and each line intersecting with at least one other line. For each of aplurality of rotational definitions, each comprising a proposed pitchand a proposed roll value, the spherical coordinates of each end pointare transformed into a Cartesian coordinate system relative to therotational definition thereby creating rotated lines, and a cumulativedeviation from a predetermined angle of the angles between intersectingrotated lines is determined. The rotational definition that has thesmallest cumulative deviation is selected as the preferred rotationaldefinition. In this example, a rotational definition is an orientationdefined by pitch and roll values.

As an alternative to the method described with respect to FIG. 35, theplane within the sphere that represents the floor could be rotated bythe rotational definition before the spherical coordinates aretransformed into locations on the rotated plane, as an alternative torotating the coordinates of each endpoint.

In other embodiments, other methods of transforming sphericalcoordinates into a Cartesian coordinate system relative to therotational definition are envisaged.

Thus, at the end of camera pose compensation process 2801, a pose-pitchand pose-roll angle have either been estimated, or obtained from themetadata in image 2908. As described with respect to FIG. 33, thepose-pitch and pose-roll values can be stored in the image metadata;alternatively, the entire image can be rotated and pose-pitch andpose-roll values of 0 stored in the metadata. These pose-pitch andpose-roll values can then be used as described with reference to FIGS.43 and 44, whenever transformations are made between spherical andCartesian coordinates.

However, in other embodiments, the camera pose compensation process maybe omitted. Knowing the camera pose means that locations in Cartesiancoordinates on a plane can be more accurately calculated, butsatisfactory results can be obtained without carrying out the camerapose compensation step.

FIG. 37

FIG. 37 details floor plan creation process 2803, in which sphericalcoordinates of points in an image in spherical format are transformedinto two-dimensional or three-dimensional Cartesian coordinates in orderto produce a floor plan of the room. As previously described, it is notnecessary for a virtual map to have been created to carry out thisprocess, as a floor plan can be created for a single room.

However, the floor plan creation process can use the connection pointsstored in virtual map 1305 to identify how adjacent rooms should bejoined together, and thereby create a floor plan of an entire building.

Process 2802 is initiated by the user selecting button 2905 on tool bar2903 at step 3701.

A cycle graph is then obtained for the room, comprising an ordered listof vertices, each representing a point selected by a user. Each vertexin the cycle graph comprises a set of spherical coordinates andcorresponding Cartesian coordinates If two planes are being considered,such as the floor and the ceiling, then each vertex comprises two setsof spherical coordinates and two sets of corresponding Cartesiancoordinates.

At step 3703 this cycle graph is used to render a floor plan of theroom. At this step, if cycle graphs exist for other rooms that haveconnection points to this room, then the cycle graphs will be renderedtogether as a plan of a building.

Step 3702 will be described further with reference to FIG. 42, whilestep 3703 will be described further with reference to FIG. 47. Beforethat, the user interface facilitating the selection of corner points bythe user will be described with reference to the illustrations in FIGS.39 to 41.

FIG. 38

The process of creating a floor plan as described with reference to FIG.37 includes the creation of a cycle graph at step 3702, which isrendered into a floor plan at step 3703. The cycle graphs created foreach room are stored as floor plan data 1308 in virtual map data 1304,while the rendered floor plans are stored as floor plans 1307.

FIG. 38 shows an exemplar method of storing cycle graphs in floor plandata 1308. In this example, the data is stored in a database structure.However it could be stored in an object-oriented programming structure,or any suitable structure.

Data 1308 includes a list 3801 of floors within the map, a list 3802 ofrooms, a list 3803 of vertices, and a list 3804 of points. Each roombelongs to a floor, and each vertex belongs to a room. Within each listof vertices for a room, an ordering field 3805 gives the position ofeach vertex, thereby creating a cycle graph for each room. Each entry inthe points list belongs to a vertex. A point includes both sphericalcoordinates 3806 and Cartesian coordinates 3807 for the point. It alsoincludes a type field 3808 indicating the type of point, for example aselected point, or an endpoint on another surface, as will be furtherdescribed. A plurality of points may belong to a vertex, depending uponwhether one or more surfaces is being considered.

Any method of storing an ordered list of selected points could be used.

FIG. 39

FIG. 39 shows interface 2901, in which image 2908 is shown in theediting area. In a similar way to camera pose compensation process 2801,input boxes 3401 are displayed allowing the user to specify the heightsof the tripod, the camera device, and the ceiling. It will be noted thatin this case an estimated value of 2.4 m has been provided for theceiling. It is not necessary to provide an exact ceiling measurement atthis stage, if the user is selecting points on the floor. If the userhas not provided an exact height, then an estimated ceiling height isused. The age of the building may be used to provide a likely height.

As a user moves cursor 3901 around editing area 2902, a cursor line 3902is displayed, running substantially upwards from the cursor point 3901.The direction of the line is vertical as reproduced in the sphericalformat. Thus, as can be seen in FIG. 39, line 3902 appears to be leaningto the left but follows the intersection between walls 3903 and 3904.

This enhanced cursor is displayed whenever floor plan creation process2803 is being used. It is achieved by transforming, in real time, theposition of the cursor in spherical coordinates into Cartesiancoordinates on the floor, as will be described with reference to FIG.43. A calculation is then made of an endpoint of the cursor line thathas the same Cartesian coordinates on the x- and y-axes but is on theceiling, as will be described further with reference to FIG. 44.

As can be seen in FIG. 39, the endpoint 3905 of line 3902 does not quitereach the ceiling. This is because the estimate of 2.4 m is, in fact,incorrect. The user may provide the actual ceiling height in the inputbox. However, for the purposes of this example, the user does not dothis, preferring to adjust the height of the ceiling after all cornersof the room have been selected, as will be described with reference toFIG. 46. As a further alternative, it is not necessary to have anaccurate measurement of the ceiling when creating a floor plan andtherefore the user may choose not to adjust it at all.

FIG. 40

The user selects a corner of the room by moving the cursor to a cornerthat is visible in image 2908, and clicking. This creates a point, forexample point 4001, which is stored in floor plan data 1308 as a vertexin list 3803, with both its spherical and Cartesian coordinates beingstored in list 3804. Coordinates for the corresponding point 4002 on theceiling plane are also stored in list 3804, referencing the same vertex.As can be seen in FIG. 40, the ceiling height is incorrect, as thecoordinates have been calculated with respect to a plane that is 2.4meters above the floor. The plane on which the cursor endpoints arelocated is referred to hereafter as the ceiling plane, as distinct fromthe ceiling of the room as visible in image 2908.

Once a point has been selected, its spherical coordinates 3806 andCartesian coordinates 3807 are stored in points list 3804. The sphericalcoordinates are used to render the selected points, and lines betweenthem, as the user moves the viewpoint of the viewer application. TheCartesian coordinates are stored in order to generate a floor plan, andare also used, as will be later described, to generate guidelines andconstrain cursor movement.

The user then moves the cursor 3901. Lines are continuously renderedbetween the current cursor position and the previously-selected corner.Thus, for example, line 4003 connects point 4001 and cursor position3901. A corresponding line 4004 joins the ceiling point 4002, and thecurrent endpoint 4005 of the cursor line. These lines, which appear tobe horizontal from the point of view of the user, can be utilised by theuser to make the selection of the next corner more accurate by liningthem up with intersections between the wall and the floor.

During the selection of the first and second corners, cursor movement isunconstrained.

FIG. 41

After the user has selected two corners, a wall is considered to befully defined. It is therefore marked with a point at each corner,vertical and horizontal bounding lines, and midpoints of the horizontallines. This can be seen for example in the selection of wall 3904. Aspreviously described, line 4101 is drawn not on the ceiling, but drawnon the ceiling plane that is 2.4 m above the floor.

After at least one wall has been completely selected, the processconstrains cursor movement, and therefore selection of points, in orderto assist with accuracy. In addition, guidelines are rendered to assistthe user.

Thus, for example, in FIG. 41, wall 3904 has been fully selected, as haswall 4102. The user is currently moving cursor 3901 to select wall 4103.The movement of cursor 3901 is constrained so that the selection of wall4103 is at one of a number of predetermined angles to wall 4102.

The Cartesian coordinates of points 4104 and 4105, defining the previouscorners, are known through the transformation method described herein.In addition, the Cartesian coordinates of the cursor 3901 are known.Therefore, an angle can be calculated between the line 4106 connectingpoints 4104 and 4105 and the line 4107 connecting point 4105 and cursorposition 3901. In order to assist the user with accurate drawing, thecursor position is constrained so that this angle is one of 90°, 120°,135°, 150°, and 180°, which are common architectural angles. If the useris mapping an unusually-constructed space, then they may wish to turnoff this constraining feature.

Guidelines are also displayed to user, for example guidelines 4108 and4109. In the example shown in FIG. 41, guideline 4108 is at 90°, in theCartesian coordinates system, to the line 4107. As can be seen, thisguideline, along with horizontal line 4107, allows the user to movecursor 3901 to a position in the corner of the room, even though thecorner is hidden behind door 4110, by lining up guideline 4108 with theintersection 4111 between the floor and wall 4112. Guideline 4109follows the same orientation in the Cartesian coordinates space, but onthe ceiling plane.

In addition, guideline 4113 connects cursor position 3901 to the firstpoint in the cycle graph, 4001. This guideline is useful when the useris closing the graph by selecting the final corner.

Additional guidelines may be displayed, for example guidelines at otherangles other than 90°.

The user continues to select corners as in the examples shown in FIGS.39 to 41 until all corners have been selected. The first corner shouldbe selected as the final corner, thus closing the cycle graph.

In the example illustrated here, the user has selected points on thefloor. However, points on the ceiling could be selected instead. Thiscould be more accurate, particularly if the room contains a lot offurniture or if there are large skirting boards or other obstructions.When the cursor point moves above the level of the camera, the processswitches to calculating Cartesian coordinates on the ceiling plane,instead of the floor plane. The cursor line then extends downwards to anendpoint calculated on a floor plane, which will only line up with theactual floor if the ceiling height is correct.

If the user is selecting points on the ceiling, it is necessary to havea reasonably accurate measurement for the ceiling height, otherwise theheight used in the calculations (the camera height subtracted from theceiling height) will be wrong and will cause inaccuracies in thetransformed Cartesian coordinates.

FIG. 42

Step 3702, at which floor plan data 1308 is obtained, is detailed inFIG. 42. At step 4201 a camera height is obtained, in the same way as atstep 3501. At step 4202 cursor movement process 2803 is initiated, whichwill be described further with respect to FIG. 43. The cursor movementprocess provides a location in Cartesian coordinates for every positionof the cursor.

At step 4203 a selection is received from the user of a point in theimage, which the user considers to be a corner. At step 4204 a questionis asked as to whether this selection closes a cycle graph, i.e. whetherit has the same spherical coordinates 3806 as the vertex in list 3803that has ordering 1 for this room. If this question is answered in thenegative, then at step 4205 a vertex with ordering n is created in list3803 for this room, and points are created in list 3804. The value of nis one more than the highest value in ordering field 3805 for this room.The points include a point having the spherical coordinates andCartesian coordinates of the selected point, and a point having thespherical coordinates and Cartesian coordinates of the corresponding endpoint; i.e. the points on both the floor and the ceiling planes.

At step 4206 a question is asked as to whether n is greater than 1, i.e.whether there is at least one previous vertex in the cycle graph. Ifthis question is answered in the affirmative, then at least two cornershave now been selected, and therefore a flag is set to constrain cursormovement at 4207.

At step 4208 a further question is asked as to whether n is greater than2, i.e. whether there are at least two previous vertices in the cyclegraph. If this question is answered in the affirmatives then there arenow at least three corners selected, and therefore a flag is set at step4209 to render guidelines. At this point, and if either of the questionsasked at 4206 or 4208 are answered in the negative, control is returnedto step 4203 and another corner selection is received. Eventually, allcorners will have been selected, the question asked at step 4204 will beanswered in the affirmative to the effect that the cycle graph isclosed.

At this point, cursor movement process 2803 is stopped at step 4210, andnormal cursor display is resumed.

Thus, at the end of step 3701, a cycle graph has been constructed byfloor plan data 1308. Each vertex in the graph comprises sphericalcoordinates showing the actual display location of the point,corresponding Cartesian coordinates on a plane, spherical coordinates ofthe opposite point on an opposite plane, and corresponding Cartesiancoordinates of the opposite point. As previously described, either thecoordinates of the floor plane or the ceiling may not match the actualfloor or ceiling in the image if the ceiling height is wrong.

FIG. 43

FIG. 43 details cursor movement process 2803. This process runs whilecorner selection step 3701 is running, and overrides the standard cursordisplay of the 360° viewer. While cursor movement process 2803 isrunning, it calculates Cartesian coordinates corresponding to thespherical coordinates of the cursor, in real time, and displays a linethat is vertical within the Cartesian coordinate system to acorresponding point on another plane within the sphere. Thus, if theuser is selecting points on the floor, the cursor line will extendupwards towards an end point on a ceiling plane, whereas if the user isselecting points on the ceiling, the cursor line will extend downwardstowards an endpoint on a floor plane. The cursor process also rendersthe lines and guidelines that are visible to the user during the cornerselection step.

The separation of the floor plan creation process 2802 and the cursormovement process 2803 described herein is only an example of how thismay be implemented, and they may in fact be the same process, or acollection of smaller processes. In addition, cursor movement process2803 is not necessary for the creation of a floor plan. If system memoryis limited, the cursor movement process may be switched off. In thiscase, when the user selected a point in the image it would be at thatpoint transformed into Cartesian coordinates, rather than the cursorposition being transformed in real time.

At step 4301 spherical coordinates within image 2908 are received bycursor movement process 2803. These would be provided by the usualcursor process of the spherical image viewer. At step 4302 the sphericalcoordinates are rotated by the pose-pitch and pose-roll values storedfor image 2908, as described with respect to FIG. 33. The image may havealready been rotated, or camera pose compensation may have been omitted,in which case the pose-pitch and pose-roll values are both zero and norotation takes place. At step 4303 the camera height calculated at step4201 and the ceiling height are retrieved from memory.

At step 4304 the spherical coordinates, as rotated at step 4302, aretransformed to Cartesian coordinates. This is done in the same way as atstep 3506 detailed with respect to FIG. 36. At step 4305 thecorresponding point on the ceiling plane or floor plane, depending onwhether the cursor is currently on the floor or on the ceilingrespectively, is calculated, as will be described further with referenceto FIG. 44. At step 4306 a cursor line is rendered on the displaybetween the cursor point and its corresponding point. The cursor line isrendered within the spherical viewer, and therefore joins the unrotatedspherical coordinates of the cursor position and its endpoint.

At step 4307 a question is asked as to whether the cycle graph hasalready been started, and if this question is answered in theaffirmative then at step 4308 lines are rendered, both on the floorplane and the ceiling plane, from the spherical coordinates of theprevious corner to the unrotated spherical coordinates of the currentcursor position, with the same happening on the other plane (the ceilingor the floor) from the spherical coordinates of the point on theopposite plane to the current cursor line endpoint.

At step 4309 a question is asked as to whether guidelines should becalculated, i.e. whether the flag to render guidelines was set at 4209.If this question is answered in the affirmative then guidelines arerendered at step 4310, as will be described with reference to FIG. 45.

At this point, and if either of the questions asked at step 4307 or step4309 is answered in the negative, then control is returned to step 4301and the next set of spherical coordinates are received. This processcontinues until cursor movement process 2803 is stopped at step 4210 offloor plan creation process 2802.

The process for constraining cursor movement is not detailed herein; thesteps to implement this include calculating, in the Cartesian coordinatesystem, the angle between the line connecting to previous points and theline from the previous point to the current cursor position, and if thatangle is within a certain distance from one of the predetermined angles,snapping the cursor to the nearest point on the line does have thatangle. This position will then be transformed back into sphericalcoordinates, rotated by necessary by the pose-pitch and pose-roll. Anadditional snap feature can be provided to close the cycle graph, suchthat the cursor position is snapped to the spherical coordinates of thefirst vertex in the graph, if it is close to those coordinates.

FIG. 44

FIG. 44 details step 4305, at which the point corresponding to thecursor point on the opposite plane is calculated. As previouslydescribed, if the pitch of the cursor point is negative, then theCartesian coordinates are calculated with respect to the floor, and thecorresponding endpoint is calculated with respect to the ceiling plane.The opposite is true if the pitch is positive. Therefore, after step4304 the spherical coordinates of the cursor point have been transformedinto three-dimensional Cartesian coordinates, where the value on thez-axis is 0 for the floor, and the current ceiling height for theceiling plane.

In order to find the point on the opposite plane (floor or ceilingrespectively) that has the same coordinates on the x-axis and they-axis, it is therefore necessary only to change the value of thez-axis, swapping 0 for the ceiling height and vice versa as required.This is done at step 4401.

At step 4402 the Cartesian coordinates obtained at step 4401 are rotatedby the inverse of the camera pose angles stored in the metadata for theimage, before these rotated Cartesian coordinates are transformed backto spherical coordinates at step 4403. Thus, both Cartesian coordinatesand spherical coordinates have been obtained for the end point of thecursor line.

It should be noted that it is the unrotated Cartesian coordinatesobtained at step 4401 that are stored in the cycle graph, as these arestored for the purposes of displaying the endpoint within the sphericalformat.

If the camera pose angles for the image were both 0, then the sphericalcoordinates of the end point could be found by merely modifying thepitch of the spherical coordinates of the cursor point, by an amountdependent on the ratio between the ceiling height and the camera height.However, this would not work if any camera pose compensation wererequired, and therefore it is preferred to rotate the Cartesiancoordinates and then transform them into spherical coordinates.

The transformation of the three-dimensional Cartesian coordinates intospherical coordinates may be carried out by any known method.

FIG. 45

FIG. 45 details step 4310 at which guidelines are rendered for thecurrent cursor point. This is carried out if the guidelines flag is set,which happens when there are at least two selected points stored in thecycle graph. It is possible to render guidelines if there is only onepoint selected, but it is preferred to only display guidelines whilecursor movement is constrained.

At step 4501 the predetermined guideline angles are retrieved. In thisembodiment, this is a single angle of 90°, but more guideline anglescould be used.

At step 4502 the first angle is selected, and at step 4503 lines arerendered on the display, which are at the selected angle in Cartesiancoordinates to the horizontal lines joining the previous selected cornerto the current cursor position. The lines are calculated in Cartesiancoordinates, and transformed to spherical coordinates for display,following a suitable rotation if camera pose angles have been stored forthe image. At step 4504 a question is asked as to whether there isanother guideline angle to consider and if so control is returned tostep 4502. Alternatively, all the guidelines have been drawn. At step4505, a final guideline is rendered, which joins the cursor position tothe position of the first corner. This is simply a line joining the twopositions in spherical coordinates.

FIG. 46

At the end of step 3702, as detailed in FIG. 42, cursor movement process2803 is stopped. A floor plan is then created at step 3703 from thefloor plan data 1308 obtained at step 3702. This step will be detailedfurther with reference to FIG. 47. The floor plan is then displayed tothe user, as shown in FIG. 46.

In this example, all the corners have been selected, with the cyclegraph being joined at point 4001. The selection of the walls istherefore visible in editing area 2902. The connection points 2906 and2907 are also visible. The floor plan created by rendering the cyclegraph is shown at 4601. It will be seen that the positions of thehotspots have been transformed into doorways in the floor plan.

Following the selection of corner points, the display in editing area2902 changes to show only the lines connecting the points, along with amidpoint of each line. This is because it is not possible to move any ofthe corner points but it is possible to alter the height of the lines.Thus, the user has chosen to select a midpoint, for example midpoint4602, and move it upwards so that line 4603 follows the intersectionbetween the ceiling and wall 3904. This has the effect of raising theceiling height for all wall selections, as can be seen in FIG. 46. Thechange in ceiling height is reflected in the input box. It is alsopossible to adjust the level of the floor using the lower midpointmarkers, for example point 4604.

While the adjustment of the height does not affect the renderedtwo-dimensional floor plan, the user may wish to adjust the height tocheck that the corners line up on the ceiling as well as the floor. Inaddition, it may be required to render a three-dimensional floor plan,in which case it would be preferred if the height of the walls wereaccurate.

When the user moves the horizontal lines using one of the midpoints,this moves the corresponding corner points upwards or downwards, andtherefore the spherical and Cartesian coordinates of the points arealtered in the points list 3804 of floor plan data 1308. For example, ifthe user moves the ceiling plane upwards within the spherical format,then all the ceiling points also move upwards and therefore theirspherical coordinates change. These replace the spherical coordinates3807 for the point in floor plan data 1308, are rotated according to thecamera pose compensation, are transformed to Cartesian coordinates, andreplace Cartesian coordinates 3807 for the point. A similar processoccurs if the floor plane is moved using one of the lower midpoints,such as midpoint 4604.

If the user wishes to create the floor plan again, then the cycle graphcan be removed by pressing DELETE button 4605. This deletes all verticesfor the room from list 3803, along with corresponding points in list3804, and returns the process to step 3702 so that the user can restartthe process of selecting corners.

If the user clicks on button 2905 then the floor plan creation process2802 is terminated.

FIG. 47

FIG. 47 details step 3703, at which the cycle graph obtained during step3701 is rendered into a floor plan. At step 4701 the floor plan data1308 is converted to a plan of the room, by plotting, on the x andy-axis of the floor plan, the Cartesian coordinates 3807 of all thepoints for the vertices in the room. At step 4702 a question is askedwhether the room has connection points stored in virtual map data 1305.If this question is answered in the affirmative then at step 4703 thefirst connection point in the room is selected and it is added to thefloor plan of the room. When making a floor plan, connection points canonly be in walls. (This is in contrast to the potential for placingconnection points in the middle of a room that was noted with respect tothe process for creating a virtual map, for example connection point212). Because each connection point must be in a wall, its horizontallocation within the wall can be calculated and therefore added to thefloor plan at step 4704.

At step 4705 a question is asked as to whether the room that theconnection point leads to is already in the floor plan of the building,i.e. whether there are vertices in list 3803 referring to this room. Ifthis question is answered in the affirmative, then at step 4706 thecurrent room and the connected room are orientated so that theconnection points are joined. At this point, and if the question askedat step 4705 is answered in the negative, then a question is asked atstep 4707 as to whether there is another connection point for the roomin the virtual map data 1305. If this question is answered in theaffirmative then control is returned to step 4703 and the nextconnection point is selected.

If all the connection points have been considered, then the floor planis rendered at step 4708. If there are no other rooms in the virtualmap, or if there are other rooms but they do not yet have cycle graphs,then the floor plan will be of a single room. If there are other cyclegraphs, then a floor plan of all the rooms will be created, oriented sothat rooms are connected at connection points.

Thus there is provided a method of creating a floor plan from aspherical image. A spherical format is created of an image obtained by acamera, the spherical format having a centre that corresponds to theposition from which the image was obtained by the camera. A firstsurface represented in the image had a first orientation and was at afirst distance from the camera when the image was obtained. A pluralityof selected points in the spherical format are obtained, and a plane isidentified that has the first orientation and that is at the firstdistance from the centre of the sphere. A plurality of locations in aCartesian coordinate system are calculated, being where lines from thecentre of the sphere to these points intersect with this plane. A floorplan is rendered using these locations.

FIG. 48

Therefore, it is possible to create a floor plan of an entire buildingby creating cycle graphs for each room in a virtual map. An example ofsuch a floor plan 4801 is shown in FIG. 48, where five rooms have beenmapped, and have been automatically orientated to connect at theirconnection points.

As previously discussed, it is possible for the cycle graphs for eachroom to be produced automatically, by performing an edge tracing processto identify corners. In this case, the entire building plan shown inFIG. 48 could be produced automatically, possibly at server 107, as soonas the user has completed creating a virtual map. In addition, floorplans of individual rooms could be automatically created, possibly atserver 107, as soon as the user uploads an image, with the floor planfor the building being created on the fly as the user creates connectionpoints as shown in FIG. 14.

After the floor plan has been rendered at step 4708, it may be displayedto a user for editing. For example, the user may wish to fine tune thefloor plan by moving walls or connection points. This may beparticularly necessary if the walls are of a greater or smallerthickness than was used during the rendering process. If the user doesmove any walls, then the Cartesian coordinates 3807 of points in list3804 will be changed, and corresponding spherical coordinates 3806calculated in a similar fashion to previously described. Thus when theuser returns to viewing rooms in the viewer, the selected corners willbe displayed according to the new spherical coordinates 3807.

In addition, the rendering process displays all connection points asstandard doors, and the user may wish to edit one or more of these toshow the actual width, door style, or orientation.

After the floor plan is rendered, and edited if necessary, it is savedin the virtual map in floor plans 1307. This may be instead of or inaddition to any floor plans generated by more conventional buttime-consuming means.

The invention claimed is:
 1. A method of transforming data provided in aspherical format, comprising the steps of, at a processor: creating aspherical format of an image obtained by a camera, wherein saidspherical format has a centre that corresponds to the position fromwhich said image was obtained by the camera, and wherein a first surfacerepresented in said image had a first orientation and was at a firstdistance from the camera when the image was obtained; obtaining aselected point in said spherical format, defined by sphericalcoordinates consisting of a yaw angle and a pitch angle defining a linefrom said centre; obtaining said first orientation and said firstdistance; identifying a plane having said first orientation that is atsaid first distance from said centre; calculating a location in aCartesian coordinate system where said plane intersects with said line,wherein two of the axes of said Cartesian coordinate system are parallelto said first plane, thereby identifying the position of the point onsaid first surface, by: calculating a relative distance from the centreof said plane to the intersection of said plane with said line, saidrelative distance being relative to said first distance, by calculatingthe tangent of either the absolute value of the pitch angle, or theabsolute value of the pitch angle subtracted from 90°, and using saidrelative distance to calculate relative coordinates in said Cartesiancoordinate system, by calculating: a) the cosine of the yaw angle,multiplied by said relative distance, and b) the sine of the opposite ofthe yaw angle, multiplied by said relative distance; and using saidlocation to transform data in said spherical image for display.
 2. Amethod according to claim 1, wherein either: a) said relative distanceis modified using said first distance before said relative coordinatesare calculated, and said relative coordinates are equal to saidlocation; or b) said relative coordinates are modified using said firstdistance to produce said location.
 3. A method according to claim 1,further comprising the step of obtaining an additional plurality ofselected points in said spherical format and calculating an additionalplurality of corresponding locations according to the method of claim 1;and wherein said step of transforming data in said spherical imagecomprises: rendering said locations as a two-dimensional plan,representing the location of said selected points on said first surface.4. A method according to claim 1, wherein a second surface representedin said image had said first orientation and was at a second distancefrom said first surface when the image was obtained, further comprisingthe steps of: identifying a second plane having said first orientationthat is at said second distance from said first plane; and identifying alocation in said Cartesian coordinate system on said second plane thatis at the shortest distance from said selected point; thereby obtainingCartesian coordinates of points on parallel planes corresponding to saidfirst and second surfaces.
 5. A method according to claim 1, wherein:said first surface is the ground as seen in said image, and said firstdistance is obtained by identifying the height of the camera from theground when the image was obtained.
 6. A method according to claim 1,further comprising the steps of: obtaining data relating to the pose ofthe camera relative to said first surface when the image was obtained,said data comprising a pose-pitch angle and a pose-roll angle; modifyingeach calculation related to a transformation between sphericalcoordinates and Cartesian coordinates and vice versa with saidpose-pitch angle and said pose-roll angle.
 7. A method according toclaim 1, further comprising the step of obtaining an additionalplurality of selected points in said spherical format and calculating anadditional plurality of corresponding locations according to the methodof claim 1; and wherein said step of transforming data in said sphericalimage comprises: calculating angles between lines joining saidlocations, and summing the deviations of these angles from apredetermined angle; and thereby estimating the deviation of the camerafrom vertical when the image was obtained.
 8. Apparatus for identifyinga location on a surface represented in a spherical image, comprising aprocessor and a memory, wherein said processor is configured to: createa spherical format of a first image obtained by a camera, wherein saidspherical format has a centre that corresponds to the position fromwhich said image was obtained by the camera, and wherein a first surfacerepresented in said first image had a first orientation and was at afirst distance from the camera when the image was obtained; obtain aselected point in said spherical format, defined by sphericalcoordinates consisting of a yaw angle and a pitch angle defining a linefrom said centre; retrieve from memory said first orientation and saidfirst distance; identify a plane having said first orientation that isat said first distance from said centre; calculate a location in aCartesian coordinate system where said plane intersects with said line,wherein two of the axes of said Cartesian coordinate system are parallelto said first plane, thereby identifying the position of the point onsaid first surface, by: calculating a relative distance from the centreof said plane to the intersection of said plane with said line, saidrelative distance being relative to said first distance, by calculatingthe tangent of either the absolute value of the pitch angle, or theabsolute value of the pitch angle subtracted from 90°, and using saidrelative distance to calculate relative coordinates in said Cartesiancoordinate system, by calculating: a) the cosine of the yaw angle,multiplied by said relative distance, and b) the sine of the opposite ofthe yaw angle, multiplied by said relative distance; and use saidlocation to transform data in said spherical image for display. 9.Apparatus according to claim 8, wherein said processor is configured toeither: a) modify said relative distance using said first distancebefore calculating said relative coordinates, wherein said relativecoordinates are equal to said location; or b) modify said relativecoordinates using said first distance to produce said location. 10.Apparatus according to claim 8, wherein said processor is furtherconfigured to obtain an additional plurality of selected points in saidspherical format and calculate an additional plurality of correspondinglocations according to the method of claim 1; and wherein said processoris configured to transform data in said spherical image by: renderingsaid locations as a two-dimensional plan, representing the location ofsaid selected points on said first surface.
 11. Apparatus according toclaim 8, wherein said processor is further configured to: identify asecond plane having said first orientation that is at a second distancefrom said first plane; and identify a location in said Cartesiancoordinate system on said second plane that is at the shortest distancefrom said selected point; thereby obtaining Cartesian coordinates ofpoints on parallel planes corresponding to said first and secondsurfaces.
 12. Apparatus according to claim 8, wherein: said firstsurface is the ground as seen in said image, and said processor isconfigured to obtain said first distance by identifying the height ofthe camera from the ground when the image was obtained.
 13. Apparatusaccording to claim 8, wherein said processor is further configured to:obtain data relating to the pose of the camera relative to said firstsurface when the image was obtained, said data comprising a pose-pitchangle and a pose-roll angle; modify each calculation related to atransformation between spherical coordinates and Cartesian coordinatesand vice versa with said pose-pitch angle and said pose-roll angle. 14.Apparatus according to claim 8, wherein said processor is furtherconfigured to obtain an additional plurality of selected points in saidspherical format and calculate an additional plurality of correspondinglocations according to the method of claim 1; and wherein said processoris configured to transform data in said spherical image by: calculatingangles between lines joining said locations, and summing the deviationsof these angles from a predetermined angle; and thereby estimate thedeviation of the camera from vertical when the image was obtained.
 15. Anon-transitory computer-readable medium with computer executableinstructions stored thereon, wherein said instructions configure aprocessor to perform the method of: creating a spherical format of animage obtained by a camera, wherein said spherical format has a centrethat corresponds to the position from which said image was obtained bythe camera, and wherein a first surface represented in said image had afirst orientation and was at a first distance from the camera when theimage was obtained; obtaining a selected point in said spherical format,defined by spherical coordinates consisting of a yaw angle and a pitchangle defining a line from said centre; obtaining said first orientationand said first distance; identifying a plane having said firstorientation that is at said first distance from said centre; calculatinga location in a Cartesian coordinate system where said plane intersectswith said line, wherein two of the axes of said Cartesian coordinatesystem are parallel to said first plane, thereby identifying theposition of the point on said first surface, by: calculating a relativedistance from the centre of said plane to the intersection of said planewith said line, said relative distance being relative to said firstdistance, by calculating the tangent of either the absolute value of thepitch angle, or the absolute value of the pitch angle subtracted from90°, and using said relative distance to calculate relative coordinatesin said Cartesian coordinate system, by calculating: a) the cosine ofthe yaw angle, multiplied by said relative distance, and b) the sine ofthe opposite of the yaw angle, multiplied by said relative distance; andusing said location to transform data in said spherical image fordisplay.